Systems and methods for contextual personalized searching

ABSTRACT

A method for providing highly relevant answers in response to a user&#39;s request uses contextual information pertaining to a user&#39;s request. A semantic entity is selected from content on the device and contextual information pertaining to the selected entity is transmitted to a server. The server constructs a query corresponding to the selected entity and translates the query into a set of simple queries; submits the simple queries to a relevant subset of a plurality of search, content and monetization sources; receives a plurality of results returned from the relevant subset of search, content and monetization sources in response to the simple queries; prioritizes the received results based on elements of the selected entity, the user, context about the selected entity, time and date, original location of the user, and aspects of the application on which the entity was selected; and presents a prioritized list of the results to the user on the client device.

RELATED APPLICATION FIELD OF THE INVENTION

This present application claims the benefit of and priority to U.S.Provisional Patent Application No. 61/512,826, entitled “SYSTEMS ANDMETHODS FOR CONTEXTUAL PERSONALIZED SEARCHING,” filed on Jul. 28, 2011,which is incorporated by reference in its entirety.

FIELD OF THE INVENTION

Embodiments of the present invention relate generally to the field ofsearching the Internet and other content sources (including, but notlimited to, advertising streams) and computerized searching. Inparticular, the systems and methods described herein allow traditionalweb, monetization, and other content to be consumed in a radicallydifferent manner based on characteristics of the consumer and his/herquery and intent.

BACKGROUND OF THE INVENTION

The World Wide Web has become a daily resource for many people. However,the large and relatively unstructured nature of the Web makes itdifficult to find sites that are useful or relevant. Similarly, thereare now many applications available to users and each application oftenhas many features, all of which can leave a user bewildered and in needof guidance. Also as users search the web and use applications theyexpect to receive relevant advertisements and/or offers.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects, features, and advantages ofthe disclosure will become more apparent and better understood byreferring to the following description taken in conjunction with theaccompanying drawings, in which:

FIG. 1A is a schematic diagram of a system in accordance with anembodiment of the present invention;

FIG. 1B is a exploded diagram of the apparatus of FIG. 1, in accordancewith an embodiment of the present invention;

FIG. 2 is a schematic diagram of a system transferring information, inaccordance with an embodiment of the present invention;

FIG. 3A is an exemplary view of a screen shot showing a webpage beforecontextual mode has been activated, according to an embodiment of thepresent invention;

FIGS. 3B and 3C are exemplary views of screen shots showing a webpageafter contextual mode has been activated and various search terms arehighlighted, according to some embodiments of the present invention;

FIG. 3D is an exemplary view of a screen shot showing an overlay windowand a webpage after a contextual search has been conducted, according toan embodiment of the present invention;

FIG. 4 is a flow diagram of a process according to an embodiment of thepresent invention;

FIGS. 5A-5D are more detailed flow diagrams of a process according to anembodiment of the present invention.

FIG. 6 is a textual representation of a data structure containinginformation regarding a sample user profile, according to an embodimentof the present invention;

FIG. 7A is an exemplary view of a screen shot showing a webpage in whicha user has highlighted a search entity;

FIG. 7B is a sample rich-query string based on the search entity andwebpage of FIG. 7A, according to an embodiment of the present invention;and

FIG. 7C is a textual representation of a data structure containing asample text search result as returned by a server that had been providedthe rich-query string of FIG. 7B, according to an embodiment of thepresent invention.

SUMMARY OF THE INVENTION

In one aspect, the present invention relates to a method for providinghighly relevant answers in response to a user's request. This isaccomplished by utilizing and applying contextual information pertainingto a user's request. A client device receives input indicating a desireto receive contextual information regarding a selected entity includedin content on the device; transmits to a server contextual informationpertaining to the selected entity; the server constructs a querycorresponding to the selected entity and translates the query into a setof simple queries; submits the simple queries to a relevant subset of aplurality of search, content and monetization sources; receives aplurality of results returned from the relevant subset of search,content and monetization sources in response to the simple queries;prioritizes the received results based on elements of the selectedentity, the user, context about the selected entity, time and date,original location of the user, and aspects of the application on whichthe entity was selected; and presents a prioritized list of the resultsto the user on the client device.

In another aspect, the present invention relates to a system forproviding highly relevant answers in response to a user's request. Thisis accomplished by utilizing and applying contextual informationpertaining to a user's request. In one particular embodiment, thepresent invention includes to a means for receiving input at a clientdevice indicating a desire to receive contextual information regarding aselected entity included in content on the device, means fortransmitting, to a server contextual information pertaining to theselected entity, means for constructing a query on the servercorresponding to the selected entity and translating the query into aset of simple queries, means for submitting the simple queries to arelevant subset of a plurality of search, content and monetizationsources, means for receiving, a plurality of search results returnedfrom the relevant subset of search, content and monetization sources inresponse to the simple queries, means for prioritizing the receivedresults based on elements of the selected entity, the user, contextabout the selected entity and user, time and date, original location ofthe user, and aspects of the application on which the entity wasselected, and means for presenting a prioritized list of the results tothe user on the client device.

DETAILED DESCRIPTION OF THE INVENTION

Various embodiments of the present invention provide a process or systemfor providing highly relevant answers in response to a user's request. Auser views a webpage, a document, or other content. According to someembodiments, the contextual search service accepts input from the userregarding one or more search terms, phrases, images, or otherinformation indicating a topic or topics of interest (hereinafterreferred to as a “search entity”). According to some embodiments,anything and everything in a document is searchable. According tovarious embodiments, the service selects additional input related to thesearch entity to determine the context of the search entity, in order toconduct a search that is more likely to be relevant than a search thatdoes not involve the context. According to various further embodiments,the service uses the search entity and context that may be related tothe search entity to search for and obtain search results. According tovarious embodiments, these results are presented to the user on theclient device.

FIG. 1A illustrates a networked environment 101 according to someembodiments, in which a contextual search service is provided. As shownin FIG. 1A, the networked environment 101 includes one or more clientmachines 102A-102N (generally referred to herein as “client machine(s)102” or “client(s) 102”) in communication with one or more servers106A-106N (generally referred to herein as “server machine(s) 106” or“server(s) 106”) over a network 104. The client machine(s) 102 can, insome embodiments, be referred to as a single client machine 102 or asingle group of client machines 102, while server(s) 106 may be referredto as a single server 106 or a single group of servers 106. Althoughthree client machines 102 and three server machines 106 are depicted inFIG. 1A, any number of clients 102 can be in communication with anynumber of servers 106. In some embodiments, a single client machine 102communicates with more than one server 106, while in another embodimenta single server 106 communicates with more than one client machine 102.In yet another embodiment, a single client machine 102 communicates witha single server 106. Further, although a single network 104 is shownconnecting client machines 102 to server machines 106, it should beunderstood that according to some embodiments, multiple, separatenetworks connect a subset of client machines 102 to a subset of servermachines 106.

In some embodiments, the computing environment 101 includes an appliance(not shown in FIG. 1A) installed between the server(s) 106 and clientmachine(s) 102. This appliance can mange client/server connections, andin some cases can load balance connections made by client machines 102to server machines 106. Suitable appliances are manufactured by any oneof the following companies (but are not limited to these companies): theCitrix Systems Inc. Application Networking Group; Silver Peak Systems,Inc, both of Santa Clara, Calif.; Riverbed Technology, Inc. of SanFrancisco, Calif.; F5 Networks, Inc. of Seattle, Wash.; or JuniperNetworks, Inc. of Sunnyvale, Calif.

Client(s) 102 and server(s) 106 may be provided as a computing device100, a specific embodiment of which is illustrated in FIG. 1B. Includedwithin the computing device 100 is a system bus 150 that communicateswith the following components: a central processing unit 121(“processor”); a main memory 122 providing non-transient storage;storage memory 128 providing non-transient storage; an input/output(I/O) controller 123; display devices 124A-124N; an installation device116; and a network interface 118. In some embodiments, the storagememory 128 includes: an operating system, software routines, and aclient agent 120. The I/O controller 123, in some embodiments, isfurther connected one or more input devices. Volatile computer memory,non-volatile computer memory, and combinations of volatile andnon-volatile computer memory are to be included within the scope ofnon-transient storage. As shown in FIG. 1B, the I/O controller 123 isconnected to a camera 125, a keyboard 126, a pointing device 127 such as(but not limited to) a mouse, and a microphone 129.

Various embodiments of the computing machine 100 include a centralprocessing unit 121 characterized by any one of the following componentconfigurations: logic circuits that respond to and process instructionsfetched from the main memory unit 122; a microprocessor unit, such as(but not limited to): those manufactured by Intel Corporation; thosemanufactured by Motorola Corporation; those manufactured by TransmetaCorporation of Santa Clara, Calif.; the RS/6000 processor such as thosemanufactured by International Business Machines; a processor such asthose manufactured by Advanced Micro Devices; or any other combinationof logic circuits. Still other embodiments of the central processingunit 122 include any one or combination of the following: amicroprocessor, a microcontroller, a central processing unit with asingle processing core, a central processing unit with two processingcores, or a central processing unit with more than one processing core.

While FIG. 1B illustrates a computing device 100 that includes a singlecentral processing unit 121, in some embodiments the computing device100 include one or more processing units 121. In these embodiments, thecomputing device 100 can store and execute firmware or other executableinstructions that, when executed, direct the one or more processingunits 121 to simultaneously execute instructions or to simultaneouslyexecute instructions on a single piece of data. In other embodiments,the computing device 100 store and execute firmware or other executableinstructions that, when executed, direct the one or more processingunits to each execute a section of a group of instructions. For example,each processing unit 121 may be instructed to execute a portion of aprogram or a particular module within a program.

In some embodiments, the processing unit 121 includes one or moreprocessing cores. For example, according to some embodiments theprocessing unit 121 has two cores, four cores, eight cores, or anynumber of cores. In some embodiments, the processing unit 121 comprisesone or more parallel processing cores. The processing cores of theprocessing unit 121, in some embodiments, access available memory as aglobal address space, or in other embodiments, memory within thecomputing device 100 is segmented and assigned to a particular corewithin the processing unit 121. In some embodiments, the one or moreprocessing cores or processors in the computing device 100 can eachaccess local memory. In still other embodiments, memory within thecomputing device 100 is shared amongst one or more processors orprocessing cores, while other memory can be accessed by particularprocessors or subsets of processors. In embodiments where the computingdevice 100 includes more than one processing unit, the multipleprocessing units are included in a single integrated circuit (IC). Thesemultiple processors, in some embodiments, are linked together by aninternal high speed bus, which may be referred to as an elementinterconnect bus.

In embodiments where the computing device 100 includes one or moreprocessing units 121, or a processing unit 121 including one or moreprocessing cores, the processors, in some embodiments, can execute asingle instruction simultaneously on multiple pieces of data (SIMD), orin other embodiments can execute multiple instructions simultaneously onmultiple pieces of data (MIMD). In some embodiments, the computingdevice 100 includes any number of SIMD and MIMD processors.

The computing device 100, in some embodiments, includes a graphicsprocessor or a graphics processing unit (not shown). In variousembodiments, the graphics processing unit includes any combination ofsoftware and hardware, and can further input graphics data and graphicsinstructions, render a graphic from the inputted data and instructions,and output the rendered graphic. In some embodiments, the graphicsprocessing unit is included within the processing unit 121. In otherembodiments, the computing device 100 includes one or more processingunits 121, such that at least one processing unit 121 is dedicated toprocessing and rendering graphics.

Some embodiments of the computing device 100 provide support for any oneof the following installation devices 116 (but is not limited to thesedevices): a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, tape drivesof various formats, USB device, a bootable medium, a bootable CD, abootable CD for GNU/Linux distribution such as KNOPPIX®, a hard-drive orany other device suitable for installing applications or software.Applications, in some embodiments, include a client agent 120, or anyportion of a client agent 120. According to some embodiments, thecomputing device 100 further includes a storage device 128 that iseither one or more hard disk drives, or one or more redundant arrays ofindependent disks; where the storage device is configured to store anoperating system, software, programs applications, or at least a portionof the client agent 120. A further embodiment of the computing device100 includes an installation device 116 that is used as the storagedevice 128.

Embodiments of the computing device 100 include any one of the followingI/O devices 130A-130N (but is not limited to these examples): a camera125, keyboard 126; a pointing device 127; a microphone 129; mice;trackpads; an optical pen; trackballs; microphones; drawing tablets;video displays; speakers; inkjet printers; laser printers; anddye-sublimation printers; touch screen; or any other input/output deviceable to perform the methods and systems described herein. According tosome embodiments, an I/O controller 123 connects to multiple I/O devices103A-130N to control the one or more I/O devices. According to someembodiments, the I/O devices 130A-130N are configured to provide storageor an installation medium 116. According to other embodiments, the I/Odevices 130A-130N are configured to provide a universal serial bus (USB)interface for receiving USB storage devices such as (but not limited to)the USB Flash Drive line of devices manufactured by Twintech Industry,Inc. Still other embodiments include an I/O device 130 that is a bridgebetween the system bus 150 and an external communication bus, such as(but not limited to): a USB bus; an Apple Desktop Bus; an RS-232 serialconnection; a SCSI bus; a FireWire bus; a FireWire 800 bus; an Ethernetbus; an AppleTalk bus; a Gigabit Ethernet bus; an Asynchronous TransferMode bus; a HIPPI bus; a Super HIPPI bus; a SerialPlus bus; a SCI/LAMPbus; a FibreChannel bus; or a Serial Attached small computer systeminterface bus.

In some embodiments, the computing machine 100 can execute any operatingsystem, while in other embodiments the computing machine 100 can executeany of the following operating systems: versions of the MICROSOFTWINDOWS operating systems such as WINDOWS 3.x; WINDOWS 95; WINDOWS 98;WINDOWS 2000; WINDOWS NT 3.51; WINDOWS NT 4.0; WINDOWS CE; WINDOWS XP;WINDOWS VISTA; and WINDOWS 7; the different releases of the Unix andLinux operating systems; any version of the MAC OS or iOS manufacturedby Apple Computer; OS/2, manufactured by International BusinessMachines; any embedded operating system; any real-time operating system;any open source operating system; any proprietary operating system; anyoperating systems for mobile computing devices; or any other operatingsystem. In still another embodiment, the computing machine 100 canexecute multiple operating systems. For example, the computing machine100 can execute PARALLELS or another virtualization platform that canexecute or manage a virtual machine executing a first operating system,while the computing machine 100 executes a second operating systemdifferent from the first operating system.

In various embodiments, the computing machine 100 is embodied in any oneof the following computing devices (but is not limited to thesedevices): a computing workstation; a desktop computer; a laptop ornotebook computer; a server; a handheld computer; a mobile telephone; aportable telecommunication device; a media playing device; a gamingsystem; a mobile computing device; a netbook; a device of the IPODfamily of devices manufactured by Apple Computer; any one of thePLAYSTATION family of devices manufactured by the Sony Corporation; anyone of the Nintendo family of devices manufactured by Nintendo Co; anyone of the XBOX family of devices manufactured by the MicrosoftCorporation; or any other type and/or form of computing,telecommunications or media device that is capable of communication andthat has sufficient processor power and memory capacity to perform themethods and systems described herein.

In other embodiments, the computing machine 100 is a mobile device suchas (but not limited to) any one of the following mobile devices: aJAVA-enabled cellular telephone or personal digital assistant (PDA),such as the i55sr, i58sr, i85s, i88s, i90c, i95cl, or the im1100, all ofwhich are manufactured by Motorola Corp; the 6035 or the 7135,manufactured by Kyocera; the i300 or i330, manufactured by SamsungElectronics Co., Ltd; the TREO 180, 270, 600, 650, 680, 700p, 700w, or750 smart phone manufactured by Palm, Inc; any computing device that hasdifferent processors, operating systems, and input devices consistentwith the device; or any other mobile computing device capable ofperforming the methods and systems described herein. In still otherembodiments, the computing device 100 is any one of the following mobilecomputing devices (but is not limited to these devices): any one seriesof Blackberry, or other handheld device manufactured by Research InMotion Limited; the iPhone manufactured by Apple Computer; Palm Pre; aPocket PC; a Pocket PC Phone; or any other handheld mobile device. Inyet still other embodiments, the computing device 100 is a smart phoneor tablet computer, such as (but not limited to) the iPhone or iPadmanufactured by Apple, Inc. of Cupertino, Calif.; the BlackBerry devicesmanufactured by Research in Motion, Ltd. of Waterloo, Ontario, Canada;Windows Mobile devices manufactured by Microsoft Corp., of Redmond,Wash.; the Xoom manufactured by Motorola, Inc. of Libertyville, Ill.;devices capable of running the Android platform provided by Google, Inc.of Mountain View, Calif.; or any other type and form of portablecomputing device.

In still other embodiments, the computing device 100 is a virtualmachine. In some embodiments, the virtual machine is any virtualmachine, such as (but not limited to) those managed by a hypervisordeveloped by XenSolutions, Citrix Systems, IBM, VMware, or any otherhypervisor. In still other embodiments, the virtual machine is managedby a hypervisor executing on a server 106 or a hypervisor executing on aclient 102.

In still other embodiments, the computing device 100 can execute,operate or otherwise provide an application that is any one of thefollowing: software; an application or program; executable instructions;a virtual machine; a hypervisor; a web browser; a web-based client; aclient-server application; an ActiveX control; a Java applet; softwarerelated to voice over internet protocol (VoIP) communications like asoft IP telephone; an application for streaming video and/or audio orreceiving and playing streamed video and/or audio; an application forfacilitating real-time-data communications; a HTTP client; a FTP client;or any other set of executable instructions. Still other embodimentsinclude a client device 102 that displays application output generatedby an application remotely executing on a server 106 or other remotelylocated machine. In these embodiments, the client device 102 can displaythe application output in an application window, a browser, or otheroutput window.

According to various embodiments, the computing device 100 furtherincludes a network interface 118 to interface to a Local Area Network(LAN), Wide Area Network (WAN), the Internet, or any type of network,through a variety of connections including, but not limited to, standardtelephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56 kb, X.25,SNA, DECNET), broadband connections (e.g., ISDN, Frame Relay, ATM,Gigabit Ethernet, Ethernet-over-SONET), wireless connections, or somecombination of any or all of the above. Connections can also beestablished using a variety of communication protocols (e.g., TCP/IP,IPX, SPX, NetBIOS, Ethernet, ARCNET, SONET, SDH, Fiber Distributed DataInterface (FDDI), RS232, RS485, IEEE 802.11, IEEE 802.11a, IEEE 802.11b,IEEE 802.11g, CDMA, GSM, WiMax and direct asynchronous connections). Insome embodiments, the network 104 comprises one or more sub-networks,and is installed between any combination of the clients 102, servers106, computing machines and appliances included within the computingenvironment 101. In some embodiments, the network 104 is: a local-areanetwork (LAN); a metropolitan area network (MAN); a wide area network(WAN); a primary network 104 comprised of multiple sub-networks 104located between the client machines 102 and the servers 106; a primarypublic network 104 with a private sub-network 104; a primary privatenetwork 104 with a public sub-network 104; or a primary private network104 with a private sub-network 104. According to some embodiments, thenetwork topology of the network 104 differs within differentembodiments, possible network topologies include: a bus networktopology; a star network topology; a ring network topology; arepeater-based network topology; or a tiered-star network topology.Additional embodiments include a network 104 of mobile telephonenetworks that use a protocol to communicate among mobile devices, wherethe protocol is any one of the following: AMPS; TDMA; CDMA; GSM; GPRSUMTS; or any other protocol able to transmit data among mobile devices.

In various embodiments, the computing environment 101 includes more thanone server 106A-106N such that the servers 106A-106N are logicallygrouped together into a server farm 106. In some embodiments, the serverfarm 106 includes servers 106 that are geographically dispersed andlogically grouped together in a server farm 106, servers 106 that arelocated proximate to each other and logically grouped together in aserver farm 106, or several virtual servers executing on physicalservers. Geographically dispersed servers 106A-106N within a server farm106 can, in some embodiments, communicate using a WAN, MAN, or LAN,where different geographic regions can be characterized as: differentcontinents; different regions of a continent; different countries;different states; different cities; different campuses; different rooms;or any, combination of the preceding geographical locations. In someembodiments, the server farm 106 is administered as a single entity,while in other embodiments, the server farm 106 includes multiple serverfarms 106.

Referring now to FIG. 2, according to various embodiments of theinvention, information 204 regarding a contextual search is transferredbetween the server(s) 106 and the component(s) 202. According to variousembodiments, the component(s) 202 is computer software. According toother embodiments, the component(s) 202 is computer hardware orcircuitry. According to various embodiments, the component(s) 202 ispart of, or executes on, the client(s) 102. According to someembodiments, the component(s) 202 is part of, or executes on, theserver(s) 106. According to various embodiments, information 204 istransferred to the server(s) 106 from the component(s) 202. According tovarious embodiments, information 204 is transferred to the component(s)from the server(s) 106. According to various embodiments, information204 is transferred back and forth between the server(s) 106 and thecomponent(s) 202.

FIGS. 3A-D illustrate the user experience of the contextual searchservice according to a non-limiting, exemplary embodiment. In FIG. 3A,the user has navigated to a webpage in a web browser using atouch-screen tablet device. In FIG. 3B, contextual mode has beenactivated, and some search entities have been identified on the displayby highlighting certain words on the underlying webpage. In FIG. 3C, theservice has automatically highlighted certain words to notify the userthat more information is available about those terms (although all theitems, words, and objects on the webpage are selectable and searchable).In FIG. 3D, the words “New” and “York” are highlighted, to the right ofthe overlay window. Although those two words are on separate lines, theservice determined that they were related by comparing those words withthe context, and then highlighting them both on the display. A searchwas conducted, which returned the results displayed in the overlaywindow.

In another embodiment, the system does not require activation ofcontextual mode to allow a user to perform a search. In theseembodiment, the user navigates to a web page, as shown in FIG. 3A andany word or series of words displayed by the web page is available forselection by the user. The user indicates a desire to search foradditional information about a term, such as by a activating a specificmouse button or by executing a “long press” on a touch screen, and theterm is highlighted in a manner similar to that shown in FIG. 3D.

FIGS. 4 and 5A-5D show non-limiting processes for a contextual searchservice, according to various embodiments of the invention. Referring toFIG. 4, in step S400, according to some embodiments, the contextualsearching mode is activated by a component(s) 202 before contextualsearches can be performed. According to some embodiments, contextualsearching mode is activated by the contextual search service acceptinginput, using a processor of a computing device, from the user to enableit. Examples include (but are not limited to), user input in the form ofmouse input, touch-screen input, keyboard input, spoken command, or thelike. For instance, in a non-limiting, exemplary embodiment, contextualsearching mode is activated with a touch-screen device by receivinginput corresponding with a sliding movement from the bottom of thescreen. For another example, in a non-limiting, exemplary embodiment,contextual searching mode is activated by receiving input from akeyboard corresponding with a user pressing Alt (or a specific metakey). Once activated, according to some embodiments, the serviceindicates, using a display, audio device, or the like, that thecontextual searching mode has been activated.

In still other embodiments, contextual mode is “always on,” that is, itdoes not need to be activated in order for contextual searches to beperformed. In these embodiments, the user selects a term to be searchedby providing specific, predefined input such as the activation of aspecific mouse button (e.g., right-clicking), the activation of aspecific mouse button in combination with a specific key of a keyboard(e.g., control-click), or by providing a specific touch gesture, such asa double-finger press, triple-finger press or long press. In someembodiments, the user's finger must contact the screen for more than 500milliseconds to register a long press. In other embodiments the lengthof time the user's finger's must contact the screen in order to registera long press is greater than 100 milliseconds, 300 milliseconds, 750milliseconds, or 1000 milliseconds.

According to some embodiments, contextual searching mode is madepersistent (that is, to stay on until it is turned off) when activated,or in some embodiments, is made optionally persistent by accepting inputindicating that a user pressed a hotkey or otherwise indicated that theoption was selected. According to further embodiments, contextualsearching mode can be made non-persistent (when it would otherwise bepersistent) by accepting input indicating that a user pressed a hotkeyor otherwise indicated that that option was selected.

For embodiments in which contextual mode is not “always on,” enteringcontextual mode involves suspending the otherwise typical user interfacebehavior of the webpage browser, email client, or document viewer. Forembodiments in which contextual mode is “always on,” the activation ofcontextual searching suspends the normal behavior of the page. Accordingto some embodiments, the ability of those software applications toaccept input as normal may be suspended until the service is no longerin contextual mode. In that case, only the service will respond to userinput directed at those applications unless the user input is directedat disabling the service or contextual mode. In these embodiments, userinteraction events with the underlying page are intercepted andredirected or, in some case, discarded. For example, the followingJavaScript event may be used to perform the interception of userinteraction events: event.stopPropagation( ).

In a non-limiting, exemplary embodiment, a computer monitor may displaythe webpage shown in FIG. 7A in a browser window, where the word “JOBS”is hyperlinked. While not in contextual mode, clicking on “JOBS” wouldcause the browser to load the webpage specified by the hyperlink. Whilein contextual mode, clicking on “JOBS” would not cause the browser toload that webpage. Instead, “JOBS” would be highlighted by the service,indicating that “JOBS” is a search entity.

Returning to FIG. 4, in step S410, according to various embodiments, thecontextual search service identifies at least one search entity.According to some embodiments, the search entity (or entities) isidentified by the contextual search service accepting input, using aprocessor of a computing device, from the user to indicate searchterm(s). Examples include (but are not limited to), user input in theform of mouse input, touch-screen input, keyboard input, or the like.According to some embodiments, the service accepts input from a mousecorresponding with a user clicking on something. According to someembodiments, the service accepts input from a touch-screen devicecorresponding with a user touching, pointing, holding, releasing, makinga dragging movement, or the like.

According to some embodiments, the contextual search service determinesthe search entity (or entities) at least in part through adisambiguation process, using a processor of a computing device. Thedisambiguation process itself also involves determining the localcontext in order to automatically select additional search terms or todetermine the boundaries of the search entity (or entities). In someembodiments, the context is used to determine how often variouscombinations of potential terms are each searched. The frequency is thenused, at least in part, to determine which additional search terms, ifany, to select, beyond the one(s) indicated by the user. According tosome embodiments, this disambiguation instead is performed by theserver(s) 106 during step S440. According to some embodiments,disambiguation takes place both during steps S410 and S440.

For example, according to a non-limiting, exemplary embodiment, atouch-screen computer user highlights the word “Depp” while viewing awebpage about the movie, The Pirates of the Caribbean. The contextualsearch service takes as input the terms surrounding Depp, which were,“The actors included Johnny Depp, Keira Knightley, and Orlando Bloom.”The service then determines that, for instance, “Johnny Depp” is a termthat is searched for more often than “Depp, Keira”. The service alsodetermines that “Johnny Depp” is a more desirable search term than“Depp”. Therefore, the service selects “Johnny Depp” as the searchentity.

According to some embodiments, the contextual search service determinesthe local context by finding other term(s), collections of term(s), orthe like, in proximity to those indicated by the user. According to someembodiments, such terms include (but are not limited to): postaladdresses, email addresses, URLs (which may or may not correspond with ahypertext link), currency, phone numbers, geographic locations, or thelike. In these embodiments, regular expressions may be used to determinethe existence of these types of terms. Exemplary regular expressionsinclude: address—[{regex: /[\t\r\n]*(.+[,\u00A0]{1,}[A-Z]{2}[\u00A0]?[\d]{5})($|[\−\r\n]+)/}, {regex:/(\d+\w*[̂\r\n,]+(?:Street|St|Avenue|Ave)(, \d+\w+Floor)?)/}];currency—{code: ‘USD’, symbol: ‘$’, name: ‘dollars?’},{code: ‘EUR’,symbol: ‘

’, name: ‘euros?’},{code: ‘GBP’, symbol: ‘£’, name: ‘pounds?’}; Twitterhandle—/̂@[A-Za-z0-9_]+$/; and phone—[{regex: /([\d]{2} [\d]{2} [\d]{2}[\d]{2} [\d]{2})/},{regex: /([\d]{4} [\d]{4})/}, regex:/((\+[\d]+)?\(?[\d]{3} \)?[−][\d]{3}[−][\d]{4})/}].

According to some embodiments, search terms may be chosen based onwhether the terms begin with a capital letter, are in all capitalletters, or are in mixed case. According to some embodiments, such termsmay be chosen from the beginning or end of a nearby sentence, where someembodiments take care to ignore periods that do not denote the end of asentence (such as those used for abbreviations such as “U.S.” or“Inc.”). Some embodiments choose such terms based on the existence ornon-existence of a nearby word “bridges” such as (but not limited to)the “of” in “the United States of America”, apostrophes used to indicatepossession (“Gerry's”), ampersands (“His & Hers”), and the like. In manyembodiments this behavior is provided via a rule base containingcommonly-used rules.

Some embodiments ignore terms that are common (for example, ignoring adate such as “February”). Some embodiments choose terms based on whetherthe surrounding HTML tag or other encoding denotes it with a special tag(for example, bold, italic, links, heading tags (<h1>, <h2>, <h3>), orthe like). Some embodiments choose terms based on whether the parentnode in a HTML or similar document is marked as a Microformat or specialsemantic tag. Some embodiments choose terms based on a Microformat wherethe Microformat consists of multiple parts (like an address), scansurrounding areas for the other related tags to highlight them at thesame time. Some embodiments choose terms based on detected sentences,dictionary terms, nouns, or the like. Various embodiments choose termsbased on any combination of the examples described above, or othersimilar examples.

In still further embodiments, every type of analysis described above isperformed in order to identify search terms extant on a displayed page.In still other embodiments, various subsets of the identified techniquesmay be applied. For embodiments in which the client machine is resourceconstrained, such as a tablet or mobile phone, a restricted area isanalyzed to conserve resources. For example, the following JavaScriptmay be used to determine if the device is a mobile device:navigator.userAgent.indexOf(‘mobile’). The rules and techniques may beapplied independently of structural boundaries in the displayed page;for example, processing is not affected by text wrapping from one lineto the next.

In step S420, according to various embodiments, the contextual searchservice notifies, using a processor, the user of the search entity (orentities) identified in step S410. According to some embodiments, thenotification is visual, such as through a display of a computer device.In some embodiments, the search entity (or entities) is highlightedwithin a document on a display. In other embodiments, a copy of theselected area is magnified in order to help the user identify theselected entity. In still further embodiments, the document is overlaidwith a darker layer to improve contrast of the selected entity. In otherembodiments, the text style of selected text is analyzed and thereplicated with the color changed to black for best contrast; in an HTMLdocument the font-style, font-family, font-weight, andtext-transformation styles will be analyzed and replicated.Alternatively, these styles can be determined using the followingJavaScript function: node.style.fontFamily.

In some embodiments, a pop-up window, text box, or the like is displayedto the user above the darker layer (if provided), which may include alist of any search entities. According to some embodiments, the selectedsearch entity (or entities) is the same as what was indicated by theuser, so no additional notification is made. According to someembodiments, the service requests input from the user regarding whichsearch entity to select. In some further embodiments, the serviceaccepts inputs from the user indicating the user's selection. In someembodiments, the input indicates that the user pressed a key, touched atouch screen, pointed with a mouse, made a voice command, or otherwiseindicated the user's selection.

In some other embodiments the selected search entity is an image, video,audio, map, or other non-textual entity. In these embodiments, ahighlight is applied to the selected entity and it is brought to theforeground, for example, displayed on top of the darker layer, ifpresent.

According to some embodiments, the service has already selected thesearch entity, whether or not textual, and requests input from the userto confirm that the user wishes to conduct a search using that searchentity.

In step S430, according to various embodiments, after a search entity isselected, that is, after the touch/click is registered and released, aquery is a string constructed and posted to a server using a processor.According to various embodiments, the rich query string is a URL encodedso that it can be included in the body of a HTTP POST request. The richquery includes both the search entity and context information(hereinafter referred to as a “rich query”). According to someembodiments, the rich query includes context information which increasesthe likelihood of a relevant search. According to some non-limiting,exemplary embodiments, the context information includes any combinationof the title of the page on which the entity is found (accessible usingthe JavaScript command, document.title), the URL of the page on whichthe entity is found (accessible using the JavaScript commanddocument.location or, in some embodiments, read from native code usingprovided APIs), META tags, information regarding word frequencies,and/or the like, of the webpage where the search entity was found. Wordfrequencies may be calculated on the client and included in the HTTPPOST request to the server. According to some embodiments, the contextinformation includes any combination of the text that is located justbefore the search entity, the text that is located just after the searchentity, the corresponding URL if hypertext tags surround the searchentity, information (such as attributes) inside a parent tag surroundingthe search entity, information regarding semantic tags surrounding thesearch entity, information regarding the application currently beingexecuted (such as a web browser), information related to previoussearches, information related to web pages visited, information relatedto hypertext links selected, information related to the user's profile,and/or the like.

Information about the application currently being executed may beexposed by an API provided by the operating system of the client. Insome particular embodiments, the system is able to access the title ofthe window, or window name, in which the selected entity appears as wellas the name of the executable application responsible for displaying theselected entity. Certain applications may allow access to furtherinformation. For example, an email program may also allow access to thetitle of the title of the email, that is, the “re” line and a wordprocessing program may allow access to the title of the displayeddocument or to other characteristics of the document, such as wordcount.

Contextual information may be obtained by traversing the DOM tree toascertain the node previous to the node containing the selected entityand extracting the content of the previous node and also to ascertainthe node following the node containing the selected entity andextracting the content of the next node. Thus, in the followingsentence: “Nicolas Sarkozy is the 23^(rd) President of the FrenchRepublic,” in which 23^(rd) President has been selected, the text“Nicolas Sarkozy” will also be extracted and included the HTTP POSTrequest. A similar technique may be used to ascertain text after theselected entity, i.e., in the sentence above the term “French Republic”may also be extracted and included in the HTTP POST request. In someembodiments, this technique is used until 25 characters before theselected entity are obtained, 25 characters after the selected entityare obtained, or both, in other embodiments it is used to obtain 50characters before the selected entity are obtained, 50 characters afterthe selected entity are obtained, or both, in still other embodiments itis used to obtain 100 characters before the selected entity areobtained, 100 characters after the selected entity are obtained, or bothand in still further embodiments it is used to obtain 250 charactersbefore the selected entity are obtained, 250 characters after theselected entity are obtained, or both. Exemplary code for extracting thecontent from the previous node is as follows:

    getPreviousTextNode: function(node) {  var childs =node.parentNode.childNodes,  idx = −1;  for (var i=0; i<childs.length;i++) {  if (childs[i] === node) {    idx = i;    break;   }  }  if(idx > 0) {   var pnode = childs[idx−1];   while(pnode.childNodes.length > 0) {    pnode =pnode.childNodes[pnode.childNodes.length−1];   }   return pnode;  } else{   return this.getPreviousTextNode(node.parentNode);  } }

Similarly, exemplary code for obtaining the content of the next node isas follows:

   getNextTextNode: function(node) { if (!node | | !node.parentNode)return null; var childs = node.parentNode.childNodes, idx = −1; for (vari=0; i<childs.length; i++) {  if (childs[i] === node) {   idx = i;  break;  } } if (idx < childs.length−1) {  var pnode = childs[idx+1]; while (pnode.childNodes.length > 0) {   pnode = pnode.childNodes[0];  } return pnode; } else {  return this.getNextTextNode(node.parentNode); } }

In some embodiments the selected entity is part of a hyperlink, or link.For example, using the example above, when selecting “23^(rd) President”the user may actually be selecting the following HTML code:

     <div>  Nicolas Sarkozy is the  <a href=‘http://www.presidents.fr’title=’Presidents of France’>   23rd President of the French Republic </a> </div>

As described above, the selection of a entity will suspend normalprocessing by the page. Thus, rather than traversing the link identifiedby the text “23^(rd) President,” that selection will be intercepted anddiscarded. The system will then recognize, using the “A” HTML tag thatthe selected entity is a link and will then also extract from the HTMLcode helpful attributes from the link, such as the HREF, ALT and TITLEattributes, in the example above. These extracted attributes areincluded with the selected entity in the HTTP POST request. The systemmay further use the JavaScript command “element.getattribute( )” inorder to acquire the attributes of a tag.

In still other embodiments, the system may acquire text associated withheaders from the page on which the selected entity appears using theJavaScript command“document.querySelectorAll(‘h1,h2,h3,h4,h5,h6,h7.h8’). The acquired textis then included in the IMP POST request sent to the server. The systemmay also search the DOM of the web page to identify META tags. The textassociated with META tags may also be useful. Each of the META tags in adocument will be returned by the following command:document.getElementsByTagName(‘meta’) and each returned tag may beanalyzed using the following exemplary code:

    for (i = 0; i < metas.length; i++) {  if(metas[i].name ==‘keywords’) {   richQuery[‘keywords’] = metas[i].content;  } }

According to a non-limiting, exemplary embodiment, FIG. 7A shows asample webpage and FIG. 7B shows a corresponding rich-query string. Theuser has highlighted “132 Crosby Street, 4^(th) Floor New York, N.Y.10012”, which is used as the search entity. The rich-query stringincludes the name of the server (“query-server-beta”), the data type(“JSON”), a requested layout type (“default”), a return type(“javascript”), a language or culture specifier (“en-US”), and the URLof the search entity's webpage (“http://kikin.com/contact”).Additionally, the rich-query string includes information found on thesearch entity's webpage, including an address (“132 Crosby Street, 4thFloor, New York, N.Y. 10012”), text in close proximity to the address(“kikin Headquarters”), a phone number (“646-395-1280”), prominent text(“Contact Us”), and keywords specified in the META tags (“kikin,facebook, twitter, social networking, price comparison, browserplugin”).

In another embodiment, information about the HTML hierarchy of the pageon which the selected entity is found may be used to determine content.In the example above, the HTML code for displaying the address to theuser may be:

     <div id=‘ address’>  Our address:  <address>   132 crosby street,New York, NY 10012  </address> </div>

In these embodiments selecting “New York” will cause the system totraverse the DOM model for the page and detect that the selected entityis part of an ADDRESS tag and a DIV tag. The extracted hierarchalinformation is included in the HTTP POST request sent to the server.

In still other embodiments, the system may identify entities usingmicroformats, when available. Microformats are attributes of HTMLelements that store extra information, not visible to the user, about atag. For example:

    <p itemprop=“address” itemscope=””itemtype=“http://data-vocabulary.org/Address”>  <spanitemprop=“stress-address”>2 Lincoln Place</span>,   <spanitemprop=“locality”>    Brooklyn   </span>   <span itemprop=“region”>   NY   </span>  </span> </p>

In the example above if the user selects “Brooklyn,” the system willdetect the presence of the property “itemprop” and parse the DOM tree totag the selection as an address. If a microformat is identified, it issaved and included in the HTTP POST request to the server.

According to various embodiments, information related the user's profilemay also be used when formulating a rich query for transmission to theserver. Such information includes, but is not limited to, anycombination of the user's history of documents visited, favoritewebsites or bookmarks, categories of favorite websites or bookmarks,shares, prior searches, recent prior searches, the search that referredthe user to the current webpage, saves, interests, social networks,frequently accessed applications/documents/websites, a Bloom filter datastructure that represents all domains that the user visited during aspecific period of time, a Bloom filter data structure that representsall domains from which the user saved a webpage, a Bloom filter datastructure that represents all domains on which the user shared awebpage, or the like. According to a non-limiting, exemplary embodiment,FIG. 6 shows a textual representation of a data structure containinginformation regarding a sample user profile.

Prior searches performed by the user are stored and may be used as partof the user profile. Additionally, as the user visits web pages, theURLs of those web pages are recorded and sent to the server. Forexample, is a user searches www.google.com for “new camera” and thenvisits multiple web pages about cameras, that search and the pagesvisited by the user will be sent to the server and used to refine acontextual query to direct it more towards camera-related results.

Similarly, the bookmarks saved by a user during web browsing areaccessed and sent to the server to be used to refine a contextualsearch. For example, if a user has bookmarked google.com, bbc.com/news,and cnn.com, the server may use this fact to provide more news-relatedsites in response to a request.

According to various embodiments, after the rich query is constructed,this rich query is sent, using a processor, from the component(s) 202 tothe server(s) 106. According to some embodiments, more than one richquery may be constructed and sent. According to further embodiments, therich queries may be identical, and in other embodiments, at least someof the rich queries may be at least partially different from anotherrich query. According to various embodiments, the processor sends therich query (or queries) to the server(s) 106 using a networking device.According to some embodiments, the rich query is not sent to a serverand is instead processed by the component(s) 202 in step S440. Accordingto further embodiments, no rich query is constructed, and instead thecontextual information that would have otherwise been used for a richquery is processed directly by the component(s) 202 in step S440.

In step S440, according to various embodiments, the contextual searchservice, using a processor, obtains the context from the rich query (orqueries) it receives from the component(s) 202. According to someembodiments, the service preprocesses the attributes stored in the richquery (or queries) to identify context that is potentially useful toimprove search results. According to some embodiments, the servicecategorizes the search entity (or entities) based on the contextinformation stored in the search query (or queries). According to someembodiments, the service disambiguates the meaning of any ambiguousterms with the help of the context information. The service may alsoleverage the results obtained by other users of the service to furtherrefine the results given to any particular user.

For embodiments in which the selected entity is an image, the image issubmitted to a facial recognition service to identify the picturedperson or product. The result from that conversion, which is text, isthen used by the service as the selected entity to search. Forembodiments in which the selected entity is audio, a digital fingerprintof the audio is submitted to an audio recognition service. In someembodiments, the audio fingerprint is generated by executing a hash onthe PCM data comprising the file. The result from that conversion, whichis text, is then used by the service as the selected entity to search.

For embodiments in which the selected entity is textual, the serveridentifies a most probable sequence of words based on the selected textand the surrounding text. In one embodiment, the server accomplishesthis by generating a list of n-word-grams. In some embodiments, thenumber words used to create the word grams are 10 words, in otherembodiments 8 words are used, in still other embodiments 6 words areused, in still further embodiments 4 words are used and in still otherembodiments 2 words are used.

As an example of this, for the sentence:

Tunisia, the country where the Arab Spring uprisings began this year,has joined the International Criminal court, becoming the first NorthAfrican country to do so

in which “international Criminal” was the selected text would result, inan embodiment in which 6 words are used, in the following set ofword-grams:

this year has joined the international year has joined the internationalcriminal has joined the international criminal court joined theinternational criminal court becoming the international criminal courtbecoming the international criminal court becoming the first criminalcourt becoming the first north year has joined the international hasjoined the international criminal joined the international criminalcourt the international criminal court becoming international criminalcourt becoming the criminal court becoming the first has joined theinternational joined the international criminal the internationalcriminal court international criminal court becoming criminal courtbecoming the joined the international the international criminalinternational criminal court criminal court becoming the internationalinternational criminal criminal court international criminal

In some embodiments, each of the generated word-grams may used as inputto a search engine, such as google.com, yahoo.com and the like. Theresults returned from these searches can be reviewed to determine if anyof the results contain links to a knowledge management site, such aswikipedia.com. Word-grams that returned search results having links to aknowledge management site are then ordered from most popular to leastpopular. The longest word-gram returning the most popular search isselected as the intended search term.

The selected search term may then be reviewed to determine if the wordor word sequence has multiple meanings. In one embodiment, this isaccomplished by consulting a directory of ambiguous phrases, such asthat maintained by wikipedia.com. In this embodiment, if the phrase isambiguous, compare the text of the page where the user originallyselected the text, with the text on each page that is linked from thedisambiguation page. Select the page with the most significant overlapand adjust the highlight text based on this. In a specific example, forthe following text: haps://www.applebank.com/aboutaspx when the userhighlights “Apple,” a request is made to a disambiguation page, e.g.,http://en.wikipedia.org/wiki/Apple_(disambiguation). The text on theabout page is compared with each page linked from above disambiguationpage. Because words used on http://en.wikipedia.org/wiki/Apple_Bankoverlap more than with the other pages, e.g., 1863 (the year the bankwas incorporated), 125^(th) (the street the first branch was locatedon), mortgage, financial, or New York, Apple Bank is selected as thedisambiguated search term to use going forward.

In step S450, according to various embodiments, the contextual searchservice, using a processor, identifies relevant content source(s) forquery submission. Content sources include, but are not limited to, theInternet, specific commerce, travel, or other service-related sources,knowledge sources such as Wikipedia and Wolfram Alpha, proprietarysources such as corporate databases, and the like. According to variousembodiments, the contextual information determined in step S440 is usedto determine which content source(s) is (or are) relevant. For example,according to some non-limiting exemplary embodiments, rich queries whichappear to relate to travel destinations (according to the search entityand its context) will be matched by the service to the Internet andtravel-related sources. According to some embodiments, the contentsource(s) is (or are) determined based at least in part on the type ofmedia that the rich query seems to be related to, for example, videos,photos, etc. According to some embodiments, the content source(s) is (orare) determined based at least in part on the ontological category thatthe rich query seems to be related to, for example, commerce, travel,etc. According to some embodiments, preferences indicated by the userand received by the service determine, in part, the type of contentsource(s).

In more detail, step S450 may be performed by searching groupings ofcontent sources, referred to herein as “vertical categories.” Ingeneral, the disambiguated word or word sequence is used to returngeneral search results which are then, in turn, compared to a verticalcategory mapping directory to identify potentially relevant contentsources. In some embodiments, the vertical categories may be selected orprioritized based on, for example, context-specific skew or userpreferences.

A specific example of this process are the steps taken to select contentsources for a general search term, “Hawaii.” A general search for“Hawaii may return, for example, the following results:

1. www.gohawaii.com2. www.gohawaii.com/big-island

3. Map of Hawaii

4. hawaii.gov/5. www.capitol.hawaii.gov/session2011/

6. Images for Hawaii

7. en.wikipedia.org/wiki/Hawaii8. www.co.hawaii.hi.us/9.9. www.lonelyplanet.com/usa/hawaii

A URL prefix to category index contains the categorization for“Map->Map”, “Images->Images”, “Wikipedia->References”, and “LonelyPlanet->Travel”. Based on these results, the serve selects the followingvertical categorization: Map, Images, References, and Travel. In oneembodiment. for each domain, the verticals most frequently accessed bythe user are tracked. That is, for example, if people onlonelyplanet.com happen to view more Images then Maps, then images willbe prioritized over maps for all users in future searches associatedwith lonelyplanet.com. However if a specific user watches videos withgreater frequency than the general population, the video category willbe introduced (or boosted) only for that user.

In step S460, according to various embodiments, the contextual searchservice, using a processor, converts the rich query (or queries) to astandard query (or queries). Standard queries are queries that areacceptable as input to the content source(s) identified in step S450.According to various embodiments, a standard query includes one or moresearch entities. According to various embodiments, a standard queryadditionally includes context information related to the search entity.According to some embodiments, if during step S440, the servicedisambiguated the meaning of any ambiguous terms with the help of thecontextual information, additional terms may be included in the standardquery, as determined by the disambiguation process. According to someembodiments, Boolean logic is applied to standard query construction. Ina non-limiting embodiment, for example, Boolean logic is used inconstructing a standard query for an Internet Search source whichaccepts Boolean search queries.

According to some embodiments, unnecessary words in the search entityare not included in the standard query. For example, if the searchentity is long, the search entity may be shortened for the standardquery, to increase the diversity of results. This is because searchstrings that are too specific may return results from too narrow a listof sources.

In greater detail, for each vertical source identified, a query stringis created that can be understood by the context source. In someembodiments, each vertical source has an associated input and outputmapping, which translates a rich query received from a client into aspecific instructions for the content source. For example, to searchYouTube the following query instruction may be generated:http://gdata.youtube.com/feeds/api/videos?q={QUERY}&orderby=relevance&safeSearch=moderate&max-results=20&format=5&v=2&client=ytapi-kikin-Youtubevideowidg-s7nshfa6-0&lr={LANGUAGE}

The placeholders {QUERY} and {LANGUAGE} would be populated based on therich query. YouTube results may be mapped in the following manner:

feed/entries/title->title

feed/entries/links/href->link

feed/entries/mediaGroup/thumbnails->thumbnail

in order to return result that can be used by the service. In someembodiments, all content sources are accessed in parallel and theresults of each source are post-filtered and re-ordered. Content andmonetization sources include, but are not limited to, on-device andin-cloud personal content; social media content; vertical categorycontent; internet and enterprise search content; local or location-basedcontent; advertisements, and products and services offers, discounts orcoupons.

In step S470, according to various embodiments, the contextual searchservice, using a processor, obtains results from the content source(s).According to various embodiments, the results are obtained by submittingthe standard query (or queries) that were constructed in step S460 tothe content source(s) identified in step S450, and the service thenaccepts results from one or more of the content source(s). According tovarious embodiments, the results are processed. According to someembodiments, the results are sorted to remove duplicates (for example,sending results to multiple content sources may return results from onecontent source that duplicate one or more of the results from anothercontent source). According to some embodiments, the results are sortedin order of contextual relevance. According to various embodiments, theresults or processed results are sent to the component(s) 202 by aprocessor, using a networking device.

According to a non-limiting, exemplary embodiment, FIG. 7C shows atextual representation of a data structure containing sample resultsreturned by a server that had been provided the rich query string ofFIG. 7B. The search entity was “132 Crosby Street, 4th Floor New York,N.Y. 10012”. Information regarding the user's profile included a countrycode, IP address, country, region, city, DMA code, region, area code,postal code, continent code, latitude and longitude. The resultsincluded map information regarding address provided in the search entity(latitude, longitude, and postal address). The results further includeda website related to the location of the address provided in the searchentity (the local government website).

Returning to FIG. 4, in step S480, according to various embodiments, thecontextual search service, using a processor, filters the results.According to some embodiments, the results are filtered according tosettings received from the user or stored in a user profile. Accordingto some embodiments, the results are filtered according to the physicallimitations of the device of the components (202), for example, thedevice screen size, latency, and/or caching.

In greater detail, the results of all content source searched may bemerged based on the order of the vertical categories and the mergedresults may be sent to the client. In some embodiments, the results aresent to client together with client-side query descriptors. In somespecific embodiments, the search results are sent to the client in aJSON-encoded structure.

In step S490, according to various embodiments, the contextual searchservice, using a processor, presents the results. According to variousembodiments, presenting the results involves rendering the results on adisplay. According to some embodiments, the service accepts input fromthe user to update the search entity so that a new search may beconducted. According to some embodiments, the results are displayed in apop-up window, dialogue box, unopened or opened tab, overlay, newpartial screen, new full screen, or the like. According to someembodiments, the results are displayed in text form; and/or rich mediaform, including audio, images, videos, maps, and/or the like; and/orapplications for displaying rich media form including audio, images,videos, maps, and/or the like.

In some embodiments, the client may itself execute parallel queries ofcontent sources and merge those results into the results received fromthe server. For these embodiments, the client follows the same querylogic as that described able in connection with the server.

Having described certain embodiments of an apparatus for interactingwith software, it will now become apparent to one of skill in the artthat other embodiments incorporating the concepts of the disclosure maybe used.

1. A method for providing contextual information regarding semanticentities included in content, the method comprising: receiving input ata client device indicating a desire to receive contextual informationregarding a selected entity included in content on the device;transmitting, to a server, metadata contextual information pertaining tothe selected entity; constructing a query corresponding to the selectedentity and translating the query into a set of simple queries;submitting the simple queries to a relevant subset of a plurality ofsearch, content and monetization sources; receiving, a plurality ofsearch results returned from the relevant subset of search, content andmonetization sources in response to the simple queries; prioritizing thereceived results based on elements of the selected entity, the user,context about the selected entity and user, time and date, originallocation of the user, and aspects of the application on which the entitywas selected; and presenting a prioritized list of the results receivedfrom the server.
 2. The method of claim 1, further comprisinghighlighting the selected entity.
 3. The method of claim 1, wherein thereceived user input is received in the form of touch-screen input. 4.The method of claim 1 wherein the received user input is received in theform of mouse input.
 5. The method of claim 1, further comprisingsuspending the default user interface behavior of the application ordevice presenting the content.
 6. The method of claim 5, furthercomprising suspending the default user interface behavior of theapplication or device presenting the content in response to user inputcomprising a dragging movement.
 7. The method of claim 1, furthercomprising gathering metadata information pertaining to the selectedentity from the content.
 8. The method of claim 1, further comprisinggathering user profile information; and transmitting, to a server,metadata information pertaining to the selected entity and the gathereduser profile information.
 9. The method of claim 1, further comprisingdisambiguating the selected entity.
 10. The method of claim 1, furthercomprising submitting the simple queries to a local or location-basedinformation source.
 11. The method of claim 1, further comprisingincluding in the received plurality of search results one or moreresults from information stored locally at the client device.
 12. Themethod of claim 1, furthering comprising filtering the result receivedfrom the submitted simple queries by removing duplicate results.
 13. Asystem for providing contextual information regarding semantic entitiesincluded in content, the method comprising: means for receiving input ata client device indicating a desire to receive contextual informationregarding a selected entity included in content on the device; means fortransmitting, to a server metadata information pertaining to theselected entity; means for constructing a query corresponding to theselected entity and translating the query into a set of simple queries;means for submitting the simple queries to a relevant subset of aplurality of search, content and monetization sources; means forreceiving, a plurality of search results returned from the relevantsubset of search, content and monetization sources in response to thesimple queries; means for prioritizing the received results based onelements of the selected entity, the user, context about the selectedentity and user, time and date, original location of the user, andaspects of the application on which the entity was selected; and meansfor presenting a prioritized list of the results received from theserver.
 14. The system of claim 13, further comprising means forhighlighting the selected entity.
 15. The system of claim 13, whereinthe means for receiving user input comprises a touch-screen.
 16. Thesystem of claim 13 wherein the means for receiving user input comprisesa mouse.
 17. The system of claim 13, further comprising means forsuspending the default user interface behavior of the application ordevice presenting the content.
 18. The system of claim 7, furthercomprising means for suspending the default user interface behavior ofthe application or device that is responsive to user input comprising adragging movement.
 19. The system of claim 13 further comprising meansfor gathering metadata information pertaining to the selected entityfrom the content.
 20. The system of claim 13 further comprising meansfor gathering user profile information.
 21. The system of claim 11,further comprising means for disambiguating the selected entity.
 22. Thesystem of claim 11, further comprising means for submitting the simplequeries to a local or location-based information source.
 23. The systemof claim 11, wherein the means for receiving the plurality of searchresults includes one or more results from information stored locally atthe client device.
 24. The system of claim 11, furthering comprisingmeans for filtering the result received from the submitted simplequeries.